Papers by Md Tahmid Rahman Laskar
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023.findings-acl)
Copied to clipboard
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, Jimmy Huang
| Challenge: | Currently, the evaluation of large language models (LLMs) such as ChatGPT in academic datasets is difficult due to the difficulty of evaluating the generative outputs produced by this model against the ground truth. |
| Approach: | They evaluate ChatGPT across 140 tasks and analyze 255K responses it generates in academic datasets. |
| Outcome: | The proposed model performs well on 140 tasks and generates 255K responses in these datasets. |
Can Large Language Models Fix Data Annotation Errors? An Empirical Study Using Debatepedia for Query-Focused Text Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Debatepedia dataset limited by noise and most queries do not have relevance to document . |
| Approach: | They harness the language generation capabilities of two LLMs to regenerate queries in a Debatepedia dataset. |
| Outcome: | The proposed model can regenerate queries from the Debatepedia dataset. |
Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspective (2023.emnlp-industry)
Copied to clipboard
| Challenge: | a study examines how to build meeting summarization systems using large language models . closed-source models are generally better in terms of performance, but open-source ones are more advantageous for industrial use . |
| Approach: | They compare closed-source and open-source meeting summarization models for real-world use . they find that closed-sourced models are generally better in terms of performance . however, smaller open-sourced LLMs could still achieve comparable performance if they are open . |
| Outcome: | The proposed model is more efficient for industrial use than closed-source models due to privacy concerns and high cost. |
Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning (2024.findings-emnlp)
Copied to clipboard
Mohammed Saidul Islam, Raian Rahman, Ahmed Masry, Md Tahmid Rahman Laskar, Mir Tafseer Nayeem, Enamul Hoque
| Challenge: | Recent studies have demonstrated that large vision language models (LVLMs) are not multi-modal and lack multi-tasking capabilities. |
| Approach: | They evaluate the performance of large vision language models (LVLMs) for chart understanding and reasoning tasks and compare them to open-source models. |
| Outcome: | The proposed models demonstrate impressive abilities in generating fluent texts covering high-level data insights, but they also encounter common problems like hallucinations, factual errors, and data bias. |
Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning? (2025.acl-industry)
Copied to clipboard
Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Ridwan Mahbub, Ahmed Masry, Mizanur Rahman, Amran Bhuiyan, Mir Tafseer Nayeem, Shafiq Joty, Enamul Hoque, Jimmy Huang
| Challenge: | Large Vision-Language Models (LVLMs) are expensive and time-consuming to evaluate . however, they are limited in their use in industrial settings due to their limited availability and limited resources. |
| Approach: | They evaluate 13 open-source LVLMs as judges for diverse chart comprehension and reasoning tasks. |
| Outcome: | The proposed models can be used to assess chart comprehension and reasoning tasks, but they are expensive and time-consuming. |
Unveiling the Essence of Poetry: Introducing a Comprehensive Dataset and Benchmark for Poem Summarization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Summarization of poetry is a challenging task as it can be easily lost if only the literal meaning is considered. |
| Approach: | They propose to use poetry as a model to summarize poetry and provide a dataset to evaluate their creative language interpretation capacity. |
| Outcome: | The proposed dataset consisting of 3011 samples and its corresponding summarized interpretation in the English language provides an opportunity to evaluate the creative language interpretation capacity of the proposed models. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
DataNarrative: Automated Data-Driven Storytelling with Visualizations and Texts (2024.emnlp-main)
Copied to clipboard
| Challenge: | Data-driven storytelling uses visual aids and visualizations to convey insights. |
| Approach: | They propose a task for data story generation using large language models and a benchmark containing 1,449 stories from diverse sources. |
| Outcome: | The proposed framework outperforms non-agentic counterparts in both model-based and human evaluations, but also reveals unique challenges in data story generation. |
Query-OPT: Optimizing Inference of Large Language Models via Multi-Query Instructions in Meeting Summarization (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing LLMs require a new call to the inference endpoint/API for each new query . repeated calls to the endpoints/AP Is expensive and impractical for many real-world use cases. |
| Approach: | They compare the performance of various LLMs for query-based meeting summarization . they find that combining queries for the same context in a single prompt can be used to minimize repeated calls. |
| Outcome: | The proposed approach reduces the number of calls to the inference endpoints/APIs in meeting summarization tasks. |
When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation (2026.eacl-long)
Copied to clipboard
Abeer Badawi, Elahe Rahimi, Md Tahmid Rahman Laskar, Sheri Grach, Lindsay Bertrand, Lames Danok, Prathiba Dhanesh, Jimmy Huang, Frank Rudzicz, Elham Dolatabadi
| Challenge: | Existing benchmarks for large language models are limited in scale, authenticity, and reliability due to the emotionally complex nature of therapeutic dialogue. |
| Approach: | They propose two benchmarks that provide a framework for evaluating large language models for mental health support. |
| Outcome: | The proposed framework provides a framework for generation and evaluation of large-scale authentic dialogue datasets and judge-reliability assessments. |
Text2Vis: A Challenging and Diverse Benchmark for Generating Multimodal Visualizations from Text (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown promise in generating visualizations from natural language, but lack of comprehensive benchmarks limits their capabilities. |
| Approach: | They propose a framework that jointly refines the textual answer and visualization code to improve GPT-4o's pass rate from 26% to 42% over direct approach. |
| Outcome: | The proposed framework increases GPT-4o’s pass rate from 26% to 42% over the direct approach and improves chart quality. |
From Charts to Fair Narratives: Uncovering and Mitigating Geo-Economic Biases in Chart-to-Text (2025.emnlp-main)
Copied to clipboard
Ridwan Mahbub, Mohammed Saidul Islam, Mir Tafseer Nayeem, Md Tahmid Rahman Laskar, Mizanur Rahman, Shafiq Joty, Enamul Hoque
| Challenge: | Existing VLMs produce more positive descriptions for high-income countries compared to middle- or low-income nations, even when country attribution is the only variable changed. |
| Approach: | They propose to automate the process by generating textual summaries of charts using vision-language models to understand how a country’s economic status influences the sentiment of generated summary. |
| Outcome: | The proposed model amplifys geo-economic biases in 6,000 chart-country pairs from six widely used vision-language models to understand how a country’s economic status influences the sentiment of generated summaries. |
DACIP-RC: Domain Adaptive Continual Instruction Pre-Training via Reading Comprehension on Business Conversations (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been used in real-world industrial scenarios for various natural language processing tasks, but their high inference cost makes their deployment impractical, necessitating the use of smaller models. |
| Approach: | They propose a continual pre-training technique that generates diverse task instructions and responses via reading comprehension on conversation transcripts, enabling better instruction generalization. |
| Outcome: | The proposed technique improves small LLMs’ domain adaptability for business conversational tasks, compared with traditional methods that rely on next-token prediction. |
Deploying Tiny LVLM Judges for Real-World Evaluation of Chart Models: Lessons Learned and Best Practices (2025.emnlp-industry)
Copied to clipboard
Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Ridwan Mahbub, Mizanur Rahman, Amran Bhuiyan, Israt Jahan, Mir Tafseer Nayeem, Shafiq Joty, Enamul Hoque, Jimmy Huang
| Challenge: | Large Vision-Language Models (LVLMs) with only 7B parameters perform poorly as judges in resource-constrained settings. |
| Approach: | They propose two approaches to ensure costefficient evaluation by combining multiple criteria into a single query and domainadaptive transfer learning to create a 2Bparameter VLM on a chart dataset. |
| Outcome: | The proposed model can effectively transfer knowledge from one dataset to another to make it a more specialized model. |
WSL-DS: Weakly Supervised Learning with Distant Supervision for Query Focused Multi-Document Abstractive Summarization (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods to generate abstractive summarizations are lacking labeled training datasets. |
| Approach: | They propose a weakly supervised approach to generate a strong summary from a set of documents based on a query. |
| Outcome: | The proposed approach sets a new state-of-the-art in terms of evaluation metrics on the Document Understanding Conferences dataset. |
BLINK with Elasticsearch for Efficient Entity Linking in Business Conversations (2022.naacl-industry)
Copied to clipboard
Md Tahmid Rahman Laskar, Cheng Chen, Aliaksandr Martsinovich, Jonathan Johnston, Xue-Yong Fu, Shashi Bhushan Tn, Simon Corston-Oliver
| Challenge: | Existing systems that align textual mentions of entities to knowledge bases are difficult to deploy in production environments. |
| Approach: | They propose a neural entity linking system that connects entities in business phone conversations to their corresponding Wikipedia and Wikidata entries. |
| Outcome: | The proposed system improves inference speed and memory consumption while maintaining high accuracy. |
AI Coach Assist: An Automated Approach for Call Recommendation in Contact Centers for Agent Coaching (2023.acl-industry)
Copied to clipboard
| Challenge: | In recent years, the utilization of Artificial Intelligence (AI) in the contact center industry is on the rise. |
| Approach: | They present a transformer-based pairwise sentence classification model that analyzes call transcripts to determine which calls are most relevant for coaching purposes. |
| Outcome: | The proposed model can determine which calls are most relevant for coaching purposes based on quality assurance queries/questions asked by managers or supervisors . |
AI Knowledge Assist: An Automated Approach for the Creation of Knowledge Bases for Conversational AI Agents (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing knowledge base is time-consuming and deters the adoption of conversational AI systems in contact centers. |
| Approach: | They propose a system that extracts knowledge in the form of question-answer (QA) pairs from historical customeragent conversations to automatically build a knowledge base. |
| Outcome: | The proposed system outperforms larger closed-source LLMs on internal data and achieves above 90% accuracy in answering informationseeking questions. |
Lost in Translation: Do LVLM Judges Generalize Across Languages? (2026.findings-acl)
Copied to clipboard
Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem, Amran Bhuiyan, Mizanur Rahman, Shafiq Joty, Enamul Hoque, Jimmy Huang
| Challenge: | MM-JudgeBench is the first large-scale benchmark for multilingual and multimodal judge model evaluation. |
| Approach: | They propose a multilingual benchmark for multilingual and multimodal judge model evaluation that includes over 60K pairwise preference instances spanning 25 typologically diverse languages. |
| Outcome: | The proposed benchmark includes over 60K pairwise preference instances spanning 25 languages. |
Improving Automatic Evaluation of Large Language Models (LLMs) in Biomedical Relation Extraction via LLMs-as-the-Judge (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models generate human-like text, making them unreliable for biomedical relation extraction tasks. |
| Approach: | They propose to use Large Language Models as judges to evaluate biomedical relation extraction . they propose structured output formatting for LLM-generated responses that helps LLMs improve their performance by 15%. |
| Outcome: | The proposed method improves LLM-Judges' performance by 15% . it is cheaper and more efficient than human evaluation metrics, the authors say . |
Aligning Text, Code, and Vision: A Multi-Objective Reinforcement Learning Framework for Text-to-Visualization (2026.eacl-long)
Copied to clipboard
| Challenge: | Text2Vis systems generate functional code but resulting charts lack semantic alignment and clarity. |
| Approach: | They propose a framework that integrates post-execution feedback with textual accuracy, code validity, and visualization quality. |
| Outcome: | The proposed framework outperforms strong zero-shot and supervised baselines and shows robust generalization to out-of-domain datasets. |
Contextualized Embeddings based Transformer Encoder for Sentence Similarity Modeling in Answer Selection Task (2020.lrec-1)
Copied to clipboard
| Challenge: | Word embeddings that consider context have attracted great attention for natural language processing tasks in recent years. |
| Approach: | They propose two different approaches to integrate contextualized word embeddings with transformer encoders for sentence similarity modeling. |
| Outcome: | The proposed model outperforms the feature-based approach on six datasets. |
ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering (2025.findings-acl)
Copied to clipboard
Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj, Firoz Kabir, Aaryaman Kartha, Md Tahmid Rahman Laskar, Mizanur Rahman, Shadikur Rahman, Mehrad Shahmohammadi, Megh Thakkar, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty
| Challenge: | Chart Question Answering systems are limited in their ability to interpret data visually and reason with visual representations. |
| Approach: | They propose a chart-based chart question-answering system that includes 1,341 charts from 99 diverse sources and 1,948 questions in various types. |
| Outcome: | The new benchmark includes 1,341 charts from 99 diverse sources and 1,948 questions in various types. |
BenLLM-Eval: A Comprehensive Evaluation into the Potentials and Pitfalls of Large Language Models on Bengali NLP (2024.lrec-main)
Copied to clipboard
Mohsinul Kabir, Mohammed Saidul Islam, Md Tahmid Rahman Laskar, Mir Tafseer Nayeem, M Saiful Bari, Enamul Hoque
| Challenge: | Large Language Models (LLMs) have emerged as one of the most important breakthroughs in natural language processing. |
| Approach: | They propose to evaluate LLMs in Bengali to benchmark their performance . they select Bangla NLP tasks such as text summarization, question answering, paraphrasing . |
| Outcome: | The proposed model performs better in some tasks than current models, but in most tasks, it is poor . |
LLM Evaluate: An Industry-Focused Evaluation Tool for Large Language Models (2025.coling-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive capability to solve a wide range of tasks in recent years. |
| Approach: | They propose to build an on-premise system for LLM evaluation to address the challenges in the evaluation of LLMs in real-world industrial settings. |
| Outcome: | The proposed evaluation system protects customer privacy and protects data integrity in real-world industrial environments. |
Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Subword tokenization approaches misalign with linguistic structure and waste capacity across languages and domains. |
| Approach: | They argue for a context-aware framework that integrates tokenizer and model co-design . they argue that tokenization should be treated as a core design problem, not an afterthought . |
| Outcome: | The proposed framework integrates tokenizer and model co-design, guided by linguistic, domain, and deployment considerations. |
Entity-level Sentiment Analysis in Contact Center Telephone Conversations (2022.emnlp-industry)
Copied to clipboard
Xue-yong Fu, Cheng Chen, Md Tahmid Rahman Laskar, Shayna Gardiner, Pooja Hiranandani, Shashi Bhushan Tn
| Challenge: | Entity-level sentiment analysis is useful in a business context to understand user emotions towards certain entities. |
| Approach: | They propose to use a model that predicts the sentiment about entities mentioned in a given text to build an entity-level sentiment analysis system that analyzes English telephone conversation transcripts. |
| Outcome: | The proposed system analyzes English telephone conversation transcripts to provide business insight. |